Skip to content

[agent-kb] Fveritas: the problem factory — a harness that generates, gates, and solves its own research problems - #280

Open
Fveritas wants to merge 25 commits into
QuantumBFS:mainfrom
Fveritas:problem-factory
Open

[agent-kb] Fveritas: the problem factory — a harness that generates, gates, and solves its own research problems#280
Fveritas wants to merge 25 commits into
QuantumBFS:mainfrom
Fveritas:problem-factory

Conversation

@Fveritas

Copy link
Copy Markdown

Team

Team name Fveritas
Members @Fveritas (solver side), @fliingelephant (generator side)

Challenge

Row
Challenge Is problem selection — not solving — the real bottleneck of autonomous research? We build a "problem factory": it mines the literature for candidate quantum many-body research problems, freezes an executable verification gate into each problem card at generation time, flies candidates through a rocket-test pipeline (static fire → hop test → verdict), and distills every outcome — including three distinct death modes — into a growing heuristics library. Goes beyond the hand-curated companion set (#124#128) by automating the curation-and-solving loop itself.
Catalog issue Addresses #133 — released by Jin-Guo Liu
Track tracks/agent-kb — the issue's Method field is "Other"; agent-kb is the track on agents that do research and the knowledge bases that ground them.

What's in this PR

Solutiontracks/agent-kb/solutions/problem-factory/ (one command: python3 run_demo.py):

Harness improvements.knowledge/models/ cards for sawtooth-chain and transverse-field-ising, a provenance-tagged sawtooth localized-magnon solvable oracle, the quantum-model missing-card rule in AGENTS.md, and the agent-kb track README.

Generator-side collaboration repo (teammate): https://github.com/Fveritas/Assembly

Fveritas added 25 commits July 27, 2026 13:55
- build_run.py materializes the gitignored challenge-run folder
  (tracks/agent-kb/results/.../run.json) so /challenge-report can find
  the sawtooth-erosion run; verified build_report.py + render_report.py
  render cleanly and the skill's scan picks up 2 completed figures
- solution README: mentor quickstart table (5 prompts, expected output
  each), requirements.txt, harness-contribution list for PR element 1
- track README: pin the chosen target (issue QuantumBFS#133, anchor QuantumBFS#112) as the
  file itself instructs
- verified end-to-end in a fresh venv: calibration CALIBRATED, demo
  1 survivor/1 deferred/3 dead, sawtooth anchors green
…4 sequential-loop repositioning

- pf/budget.py + pf/round2.py + run_learning_loop.py: round-2 fleet licensed
  by round-1 heuristics; hop budget waste 29% -> 0%, zero deaths, deferred
  card relaunched bigger with next-step recommendation
- tests/test_learning_loop.py anchors (fleet licensing, fingerprint
  non-collision, static fire, round-1 waste anchor 18/63)
- docs/design/problem-generation.md: v3 methodology (badness filter +
  decomposition granularity) + sequential decision loop addendum
- docs/discussion/: Day-4 brainstorm log (repositioning rationale)
- README: 'The loop closes' section + quickstart note; log.md Day 3/4
…ors), deferred verdict, QMC routing

- cards/round3/tfim-ratio-sqrt5-001.yaml: frozen gate (|R-sqrt5|/sigma_R, kill 2.0)
- pf/tfim2d.py: chain/square/triangular/honeycomb PBC clusters, even-parity ED
- tests/test_tfim2d.py: dimer exact, classical limit, strong-field bound,
  parity, Jordan-Wigner cross-check (1e-10), Binder limits — TDD, written first
- round 1 died no_solver (registry gate), builder added, round 2 deferred (0.73):
  R = 2.186 +/- 0.068 vs sqrt5; sigma_R 5700x too large -> routed to sign-free QMC
- brief + data + figure + 2 heuristics entries + telemetry both rounds
- INTERFACE.md: tfim_2d registered
…clean-room reproduction verified

- ARGUMENT.md: single human-readable doc arguing correctness (Bethe/XDiag
  anchors, blind calibration 5/5+3/3, frozen gates) and usefulness (teeth in
  both directions, learning loop 29%->0% waste, held-out generalization)
- README: link ARGUMENT.md from the top for reviewers
- reproduction verified in a fresh clone + fresh venv: all 5 README prompts
  green, including the optional learning loop
results/ stays out of git per submission rules, so copy the self-contained
report.html (figures embedded as data URIs) + report.json + run.json into
solutions/problem-factory/report/ and link it from the README top line.
@Fveritas

Copy link
Copy Markdown
Author

Reproducible prompt (challenge-report element) — from a fresh clone, each numbered line is one short prompt to the agent, or run the command directly. Full version with expected outputs: tracks/agent-kb/solutions/problem-factory/README.md § Reproduce.

python3 -m pip install -r tracks/agent-kb/solutions/problem-factory/requirements.txt
cd tracks/agent-kb/solutions/problem-factory
# prompt runs expect
1 "Run the problem factory's first flight" python3 run_demo.py (~2 s) launched 5: survivor 1, deferred 1, dead 3
2 "Run the calibration gate" python3 run_calibration.py (<1 s) -> CALIBRATED (dev 5/5 pos, test 1/1 pos)
3 "Reproduce the issue #112 sawtooth solve" python3 tests/test_sawtooth.py && python3 run_sawtooth.py (~20 s) all anchors green + two figures in briefs/figures/
4 "Build the challenge run folder" python3 build_run.py (<1 s) tracks/agent-kb/results/20260728-sawtooth-erosion/run.json
5 "Generate the challenge report" /challenge-report interactive report → report.html next to run.json

Optional: close the learning loop — python3 tests/test_learning_loop.py && python3 run_learning_loop.py (~10 s) replays round 1, regenerates a heuristics-licensed round-2 fleet, and reports budget waste dropping 29% → 0%.

Verified on this branch before opening the PR: all tests/ green, calibration CALIBRATED, demo reruns byte-identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant